Graphics processing unit and information processing apparatus

ABSTRACT

According to one embodiment, a graphics processing unit detects characteristics of video data by analyzing frames of the video data by using at least one first processing core of a plurality of processing cores, and applies a process, which is associated with the detected characteristics of the video data, to audio data on a memory, by using at least one second processing core of the plurality of processing cores. The graphics processing unit includes an audio signal output interface and outputs an audio signal corresponding to the audio data, to which the process has been applied, to a sound device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2010-150457, filed Jun. 30, 2010; theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a graphics processingunit and an information processing apparatus using the graphicsprocessing unit.

BACKGROUND

In general, in an information processing apparatus such as a personalcomputer, a graphics processing unit (GPU) is used in order to enhancethe image-rendering capability of the information processing apparatus.Most of modern GPUs include multiple processing cores which are operablein parallel, and can execute, for example, 2D or 3D graphics arithmeticoperations at high speed.

On the other hand, the processing of audio data (also referred to as“sound data”) in the information processing apparatus is usuallyexecuted by a digital signal processor (DSP) which is called “soundcontroller”. However, in general, the arithmetic processing capabilityof the DSP is relatively low. It is difficult, therefore, for the DSPalone to execute a high-level process on the audio data.

In addition, recently, a media processing LSI has been developed, whichis designed to handle both video data and audio data.

However, what is realized by an ordinary media processing LSI is, atmost, a function of encoding or decoding video data and a function ofencoding or decoding audio data. Besides, in the ordinary mediaprocessing LSI, a process on video data and a process on audio data areindependent from each other.

In digital content (e.g. movie, DVD video, broadcast program, etc.)including video data and audio data, the video data and audio data arecorrelated. Therefore, in an apparatus which plays back digital content,a novel function needs to be realized for making the process on videodata and the process on audio data cooperate with each other.

BRIEF DESCRIPTION OF THE DRAWINGS

A general architecture that implements the various features of theembodiments will now be described with reference to the drawings. Thedrawings and the associated descriptions are provided to illustrate theembodiments and not to limit the scope of the invention.

FIG. 1 is an exemplary perspective view illustrating the structure of aninformation processing apparatus according to an embodiment;

FIG. 2 is an exemplary block diagram illustrating the systemconfiguration of the information processing apparatus of the embodiment;

FIG. 3 is an exemplary block diagram illustrating the structure of agraphics processing unit which is provided in the information processingapparatus of the embodiment;

FIG. 4 is an exemplary diagram for describing the flow of a processwhich is executed by the graphics processing unit shown in FIG. 3;

FIG. 5 is an exemplary diagram for describing the flow of areverberation process which is executed by the graphics processing unitshown in FIG. 3;

FIG. 6 is an exemplary diagram for describing the flow of a processwhich is executed on video data and audio data by the graphicsprocessing unit shown in FIG. 3;

FIG. 7 is an exemplary diagram for describing an example of an audioprocess which is executed by the graphics processing unit shown in FIG.3, based on an analysis result of video data;

FIG. 8 is an exemplary diagram for describing the flow of video data andaudio data in a system using a conventional graphics processing unit;

FIG. 9 is an exemplary diagram for describing the flow of video data andaudio data in a system using the graphics processing unit shown in FIG.3; and

FIG. 10 is an exemplary diagram for describing the functional structureof the graphics processing unit shown in FIG. 3.

DETAILED DESCRIPTION

Various embodiments will be described hereinafter with reference to theaccompanying drawings.

In general, according to one embodiment, a graphics processing unitcomprises a host interface, a memory controller, a plurality ofprocessing cores, an arithmetic control unit, a video signal outputinterface, and an audio signal output interface. The host interface isconfigured to receive video data and audio data from a host. The memorycontroller is configured to store the received video data and audio datain a memory. The plurality of processing cores are coupled to thememory. The arithmetic control unit is configured to detectcharacteristics of the video data by analyzing frames of the video dataon the memory by using at least one first processing core of theplurality of processing cores, and to apply a process, which isassociated with the detected characteristics of the video data, to theaudio data on the memory, by using at least one second processing coreof the plurality of processing cores. The video signal output interfaceis configured to generate a video signal corresponding to the video dataand to output the generated video signal to a display device. The audiosignal output interface is configured to generate an audio signalcorresponding to the audio data to which the process has been applied,and to output the generated audio signal to a sound device.

FIG. 1 is a perspective view showing the external appearance of aninformation processing apparatus according to an embodiment. Theinformation processing apparatus is realized, for example, as anotebook-type personal computer 10. As shown in FIG. 1, the computer 10comprises a computer main body 11 and a display unit 12. An LCD (liquidcrystal display) 17 is built in the display unit 12. The display unit 12is attached to the computer main body 11 such that the display unit 12is rotatable between an open position where the top surface of thecomputer main body 11 is exposed, and a closed position where the topsurface of the computer main body 11 is covered.

The computer main body 11 has a thin box-shaped housing. A keyboard 13,a power button 14 for powering on/off the computer 10, an inputoperation panel 15, a touch pad 16, and speakers 18A and 18B aredisposed on the top surface of the housing of the computer main body 11.Various operation buttons are provided on the input operation panel 15.

The right side surface of the computer main body 11 is provided with aUSB connector 19 for connection to a USB cable or a USB device of, e.g.the USB (universal serial bus) 2.0 standard. In addition, an externaldisplay connection terminal (not shown), which supports, e.g. the HDMI(high-definition multimedia interface) standard, is provided on the rearsurface of the computer main body 11. The external display connectionterminal is used to output a digital video signal to an externaldisplay.

FIG. 2 is a view showing the system configuration of the computer 10.

The computer 10, as shown in FIG. 2, comprises a CPU 101, a north bridge102, a main memory 103, a south bridge 104, a graphics processing unit(GPU) 105, a memory 105A, a BIOS-ROM 107, a LAN controller 108, a harddisk drive (HDD) 109, an optical disc drive (ODD) 110, a USE controller111A, a card controller 111B, a wireless LAN controller 112, an embeddedcontroller/keyboard controller (EC/KBC) 113, and an EEPROM 114.

The CPU 101 is a processor for controlling the operations of therespective components in the computer 10. The CPU 101 executes anoperating system (OS) 201 and various application programs, which areloaded from the HDD 109 into the main memory 103. The applicationprograms include a video player for playing back various digitalcontents (e.g. movie, DVD video, broadcast program data, etc.). Thevideo player is configured to play back video data and audio data whichare included in digital contents by using the GPU 105. The CPU 101 alsoexecutes a BIOS which is stored in the BIOS-ROM 107. The BIOS is aprogram for hardware control.

The north bridge 102 is a bridge device which connects a local bus ofthe CPU 101 and the south bridge 104. The north bridge 102 includes amemory controller which access-controls the main memory 103. The northbridge 102 also has a function of communicating with the GPU 105 via,e.g. a PCI EXPRESS serial bus.

The GPU 105 controls the LCD 17 that is used as a display monitor of thecomputer 10, and an external display. A video signal, which is generatedby the GPU 105, is sent to the LCD 17. In addition, the GPU 105 can senda digital video signal to an external display 1 via an HDMI controlcircuit 3 and an HDMI terminal 2. The HDMI terminal 2 is theabove-described external display connection terminal. The HDMI terminal2 is capable of sending a non-compressed digital video signal and adigital audio signal to the external display 1, such as a TV, via asingle cable. The HDMI control circuit 3 is an interface for sending adigital video signal to the external display 1, which is called “HDMImonitor”, via the HDMI terminal 2.

The GPU 105 is coupled to the CPU 101 via, e.g. a PCI Express serialbus. The GPU 105 is configured to process graphics data, video data andaudio data in accordance with a request from the CPU 101. To be morespecific, the GPU 105 includes an audio signal output interface (audiosignal output interface unit) for outputting an audio signal (alsoreferred to as “sound signal”) to a speaker or some other external sounddevice, and a function of analyzing and processing audio data.Accordingly, in the computer 10, not only graphics data, but also bothvideo data and audio data can be processed by the GPU 105. A videosignal and an audio signal (also referred to as “sound signal”), whichare generated by the GPU 105, are directly output from the GPU 105 to adisplay device (e.g. LCD 17, external display) and a sound device(speaker 18A, 18B, external sound device).

In this manner, in the present embodiment, the GPU 105 cansimultaneously process both video data and audio data, and can directlyoutput to the outside a video signal corresponding to video data and anaudio signal corresponding to audio data.

The GPU 105 incorporates a plurality of processing cores (streamingprocessors). The GPU 105 receives audio data from the CPU 101 via, e.g.a PCI EXPRESS bus, and processes the received audio data by using atleast one of the processing cores. The process of the audio data by theprocessing core may be realized by controlling an instruction setcorresponding to the processing core by firmware or software. Thereby,the process of audio data, which is executed by the GPU 105, can be madeprogrammable, and various processes (edit) on audio data can be executedby the GPU 105. The edit of audio data means applying a process to audiodata. The edit of audio data includes, for instance, various soundeffects, various filters, 2D to 3D sound conversion, and volume control.For example, the GPU 105 can analyze audio data according to a programloaded on the processing cores, and can adaptively process (edit) theaudio data according to the analysis result.

Similarly, the GPU 105 can also receive video data from the CPU 101 via,e.g. a PCI EXPRESS bus, and can process the received video data by usingat least one of the processing cores. The process of the video data bythe processing core may also be realized by controlling an instructionset corresponding to the processing core by firmware or software.Thereby, the process of video data, which is executed by the GPU 105,can be made programmable, and various processes (edit) on video data canbe executed by the GPU 105. The edit of video data means applying aprocess to video data. The edit of video data includes, for instance,high-resolution conversion, 2D to 3D video conversion, resize, androtation. The GPU 105 can analyze each of frames of video data accordingto a program loaded on the processing cores, and can adaptively process(edit) the video data according to the analysis result.

Furthermore, the GPU 105 can adaptively process audio data in accordancewith the content of video data. In this case, the GPU 105 detectscharacteristics of video data by analyzing frames of the video data, andapplies a process, which is associated with the detected characteristicsof video data, to the audio data. For example, the GPU 105 detects,based on the characteristics of video data, a specific scene in thevideo data (e.g. a conversation scene, a dark-image scene, a light-imagescene, a fast-moving scene, a slow-moving scene, etc.), and applies apredetermined process, which is associated with the specific scene, toeach of frames in the audio data belong to the detected specific scene.In the analysis of each video frame, for example, the GPU 105 maygenerate a histogram of each frame of the video data, and may detect thecharacteristics of each video frame, based on the histogram.

Conversely, the CPU 105 can detect the characteristics of audio data byanalyzing frames of the audio data by using at least one processingcore, and can apply a process, which is associated with the detectedcharacteristics of the audio data, to video data by using at least oneother processing core.

The processing result of video data and the processing result of audiodata may be directly output to the outside as a video signal and anaudio signal. Alternatively, the processing result of video data and theprocessing result of audio data may be transferred from the GPU 105 tothe CPU 101. Besides, in order to recursively execute the process ofvideo data and the process of audio data, the processing result of videodata and the processing result of audio data may be fed back to theassociated processing cores in the GPU 105.

Moreover, the GPU 105 may add a time stamp (time code) to each videoframe in video data and each audio frame in audio data. By the timestamp, processed video data and processed audio data can besynchronized, and a video signal corresponding to the processed videodata and an audio signal corresponding to the processed audio data canbe output to the outside in the state in which the video signal andaudio signal are synchronized.

The south bridge 104 controls devices on a PCI (Peripheral ComponentInterconnect) bus and devices on an LPC (Low Pin Count) bus. The southbridge 104 includes an IDE (Integrated Drive Electronics) controller forcontrolling the HDD 109 and ODD 110.

The LAN controller 108 is a wired communication device which executeswired communication of, e.g. the IEEE 802.3 standard. On the other hand,the wireless LAN controller 112 is a wireless communication device whichexecutes wireless communication of, e.g. the IEEE 802.11g standard. TheUSB controller 111A communicates with an external device which supports,e.g. the USB 2.0 standard (the external device is connected via the USBconnector 19). The card controller 111B executes data write and dataread in/from a memory card such as an SD card, which is inserted in acard slot provided in the computer main body 11.

The EC/KBC 113 is a one-chip microcomputer in which an embeddedcontroller for power management and a keyboard controller forcontrolling the keyboard 13 and touch pad 16 are integrated. The EC/KBC113 has a function of powering on/off the computer 10 in accordance withthe user's operation of the power button 14.

Next, referring to FIG. 3, the structure of the GPU 105 is described.

The GPU 105 comprises a PCI Express interface 201, a hub 202, a graphicsengine 203, a plurality of streaming processors (SP) 204, a memorycontroller 205, memory 105A, a video signal output interface 206, and anaudio signal input/output interface 207.

The PCI Express interface 201 is a host interface (host interface unit)for communicating with a host (CPU 101, main memory 103). The PCIExpress interface 201 receives graphics data, video data and audio datafrom the host (CPU 101, main memory 103). In addition, the PCI Expressinterface 201 receives, from the host, requests (instructions) forprocessing graphics data, video data and audio data. The graphics data,video data and audio data, which have been received from the CPU 101 bythe PCI Express interface 201, are stored in the memory 105A by thememory controller 205 (buffering). The memory 105A is a memory called“frame buffer” or “video memory”. The memory 105A is used as a localmemory of the GPU 105. The memory 105A is realized by, for example,either an external memory which is connected to the GPU 105 via a memorybus, or an internal memory which is built in the GPU 105, or both of theexternal memory and the internal memory. The internal memory may be acache memory.

The graphics engine 203 executes processes on the graphics data, videodata and audio data by using the streaming processors (SP) 204. Eachstreaming processor (SP) 204 functions as a processing core (alsoreferred to as “programmable shader”) which can execute a graphicsarithmetic operation and a general-purpose arithmetic operation. Thecontent of the arithmetic operation, which is executed by each streamingprocessor (SP) 204, can be determined by instructions which are loadedfrom the graphics engine 203 in each streaming processor (SP) 204. Eachstreaming processor (SP) 204 can refer to the graphics data, video dataand audio data on the memory 105A via the memory controller 205.Further, each streaming processor (SP) 204 can write a processing resultof the graphics data, video data or audio data in the memory 105A viathe memory controller 205.

Processes to be executed are allocated to the respective streamingprocessors (SP) 204 by the graphics engine 203. In this sense, thegraphics engine 203 is a dispatcher (also referred to as “scheduler”)which allocates the plural processes (tasks) to the plural streamingprocessors (SP) 204, thereby to execute the plural processes inparallel.

In the present embodiment, the graphics engine 203 includes anarithmetic control unit 203A, in order to make it possible to adaptivelyprocess each frame of audio data in accordance with an analysis resultof each frame of video data.

The arithmetic control unit 203A allocates to at least one firststreaming processor (SP) 204 of the plural streaming processors (SP) 204a first task for detecting the characteristics of video data byanalyzing the frames of the video data on the memory 105A. This task maybe realized as a thread. Further, the arithmetic control unit 203Aallocates to at least one second streaming processor (SP) 204 of theplural streaming processors (SP) 204 a second task for applying aprocess, which is associated with the detected characteristics of videodata, to the audio data on the memory 105A. The first task and thesecond task are executed in parallel by different streaming processors(SP) 204.

Each first streaming processor (SP) 204, to which the first task isallocated, executes instructions which are loaded from the arithmeticcontrol unit 203A into the first streaming processor (SP) 204. Eachfirst streaming processor (SP) 204 fetches a frame of video data fromthe memory 105A, and analyzes the image of the frame. In the analyzingprocess, it is possible to execute, for example, a process of detectingimage characteristics of each video frame, a process of detecting aspecific object (e.g. a person) appearing in the video frame, a processof detecting a movement of the specific object, and a process ofdetecting the position of the specific object in the video frame. Theanalysis processing result indicative of the characteristics of eachvideo frame is stored in the memory 105A.

Each second streaming processor (SP) 204, to which the second task isallocated, executes instructions which are loaded from the arithmeticcontrol unit 203A into the second streaming processor (SP) 204. Eachsecond streaming processor (SP) 204 applies to each audio frame aprocess (edit) which is associated with the characteristics of thecorresponding video frame, for example, based on the analysis processingresult indicative of the characteristics of each video frame, which isstored in the memory 105A. In this case, each second streaming processor204 fetches a frame of audio data from the memory 105A, and applies aprocess (edit), which is associated with the characteristics of thecorresponding video frame, to the fetched frame of audio data.

In the present embodiment, as described above, it is possible to executein parallel the process of analyzing the video data on the memory 105Aand the process of applying a process (edit) to the audio data on thememory 105A in accordance with the analysis result of the video data, byusing a plurality of streaming processors (SP) 204.

The video signal output interface (video signal output interface unit)206 generates a video signal corresponding to the video data which isprocessed by the streaming processor (SP) 204, and outputs the generatedvideo signal to the display device. The video signal output interface206 includes a digital-to-analog converter (DAC) 301, a display analogoutput interface (display analog output interface unit) 302, and adisplay digital output interface (display digital output interface unit)303. The digital-to-analog converter (DAC) 301 and display analog outputinterface 302 generate an analog video signal (RGB signal) from theprocessed video data. The analog video signal (RGB signal) is output toan external RGB monitor via an analog video signal output pins which areprovided in the GPU 105. The display digital output interface 303generates a digital video signal from the processed video data. Thedigital video signal is output to an external digital monitor (DVImonitor, HDMI monitor, DP monitor, etc.) via a digital video signaloutput pins which are provided in the GPU 105.

The audio signal input/output interface (audio signal input/outputinterface unit) 207 receives an audio signal from an external device andoutputs an audio signal to the external device. The audio signalinput/output interface 207 includes a digital-to-analog converter (DAC)311, a sound analog output interface (sound analog output interfaceunit) 312, an analog-to-digital (ADC) 313, a sound analog inputinterface (sound analog input interface unit) 314, a sound digitaloutput interface (sound digital output interface unit) 315, and a sounddigital input interface (sound digital input interface unit) 316.

The digital-to-analog converter (DAC) 311 and sound analog outputinterface 312 generate an analog audio signal (also referred to as“analog sound signal”) from processed audio data. The analog audiosignal is output to the speakers or some other external analog sounddevice via an analog audio signal output pins which are provided in theGPU 105. The sound analog input interface 314 inputs an analog audiosignal from a microphone or the like, and the analog-to-digitalconverter (ADC) 313 converts the input analog audio signal to a digitalaudio signal.

The sound digital output interface 315 generates a digital audio signal(also referred to as “digital sound signal”) from processed audio data.The digital audio signal is output to an external digital sound devicevia a digital audio signal output pins which are provided in the GPU105. The sound digital input interface 316 inputs a digital audio signalfrom the outside.

The input audio signal can also be processed by the streaming processors(SP) 204. For example, the arithmetic control unit 203A can also executea speech recognition process of recognizing a speech signal (an audiosignal), which is input from a microphone, by using at least onestreaming processor (SP) 204. The result of speech recognition can besent to the host via the PCI Express interface 201.

In the embodiment, as described above, the GPU 105 is additionallyprovided with external interfaces which are necessary forinputting/outputting audio signals. Moreover, the high-level audioprocess, in which an audio process that is conventionally implemented inDSPs is expanded, is executed by using the memory 105A and streamingprocessors 204 of the GPU 105. Thereby, a DSP for processing audio canbe excluded from the system of the computer 10, and the total cost ofthe system can be reduced.

FIG. 4 illustrates an arithmetic process which is executed by thestreaming processor (SP) 204. At least one instruction from thearithmetic control unit 203A is input to the streaming processor 204(Input Assembler). The streaming processor (SP) 204 fetches data fromthe memory 105A and executes various arithmetic operations on thefetched data, in accordance with the instruction. The content of thearithmetic operation that is executed is determined by the instruction.The streaming processor (SP) 204 successively executes the fetch andarithmetic operation of data, while writing the arithmetic result in thememory 105A or feeding back the arithmetic result to the streamingprocessor (SP) 204 itself, according to the necessity.

FIG. 5 illustrates an example of an arithmetic process for areverberation process which is executed by the streaming processor (SP)204. The following instruction is preloaded in the streaming processor(SP) 204:

data[n]=data[n]+data[n−1]×R

where R is a coefficient representing the intensity of reverberation inthe reverberation process.

Audio data comprises a series of data (A, B, C, D, . . . ) which arearranged in a time-series manner. The audio data is buffered in thememory 105A in advance. The streaming processor (SP) 204 acquires theaudio data as discrete data from the memory 105A, and executes aninstruction which is given in advance. Thereby, audio data, in which areverberation effect is applied to each discrete data, is generated atlast by the streaming processor (SP) 204. In a part “Output” in FIG. 5,the GPU 105 outputs to the outside an audio signal corresponding to theaudio data which has been subjected to the reverberation process, thatis, the audio data to which the reverberation effect has been applied.

In the above-described manner, a substitute process for the DSP can beexecuted by the streaming processor (SP) 204 in the GPU 10.

Next, referring to FIG. 6, a description is given of the flow of aprocess on video data and audio data, which is executed by the GPU 105.In the present embodiment, all of the following processes can becompleted by the GPU 105.

(1) The GPU 105 receives video data and audio data from the host, andstores the received video data and audio data in the memory 105A. Inaddition, the GPU 105 may receive, from the host, streaming data inwhich video data and audio data are multiplexed, and may store thereceived streaming data in the memory 105A.

(2) The GPU 105 analyzes the video data and processes the video data inaccordance with the analysis result of the video data. The analysis andprocessing of the video data are executed by at least one streamingprocessor (SP) 204 in the GPU 105. The analysis and processing of thevideo data by the streaming processor (SP) 204 are executed according toinstructions loaded in the streaming processor (SP) 204. In parallelwith the analysis and processing of the video data, the GPU 105 cananalyze the audio data and process the audio data in accordance with theanalysis result of the audio data. The analysis and processing of theaudio data are executed by at least one other streaming processor (SP)204 in the GPU 105. The analysis and processing of the audio data by thestreaming processor (SP) 204 are executed according to instructionsloaded in the streaming processor (SP) 204.

(3) In the processing of the video data, an arithmetic result of animmediately preceding video frame may be fed back for the purpose of anarithmetic operation of the present video frame. Similarly, in theprocessing of the audio data, an arithmetic result of an immediatelypreceding audio frame may be fed back for the purpose of an arithmeticoperation of the present audio frame. Further, as described above, basedon the analysis result of each of the video frames, the GPU 105 canprocess the corresponding audio frame.

(4) The GPU 105 directly outputs to the outside a video signalcorresponding to the processed video data and an audio signalcorresponding to the processed audio data.

Thereby, after the data is once loaded in the memory 105A of the GPU105, the GPU 105 does not need to communicate with the CPU 101.Therefore, the CPU resources can be reduced.

Next, referring to FIG. 7, a description is given of an example in whichan audio frame is processed based on an analysis result of a videoframe. The case is now described, by way of example, in which each offrames in audio data belonging to a scene of a person who is speaking isso processed as to emphasize the voice of the person so that the voicemay be easily heard.

In the streaming processor (SP) 204, instructions to execute thefollowing processes are preloaded.

(1) An image of each frame of video data is analyzed, the lips aresearched, and the motion of the lips is recognized.

(2) From the motion of the lips, a “scene in which a person is speaking”is detected (recognized).

(3) A process of gating, equalizing, etc. of a vocal-cords range of aperson is executed on each of the frames of the audio data belonging tothe scene in which the person is speaking, so that the vocal-cords rangemay be emphasized.

(4) Each of the frames of the processed audio data is fed back to thecorresponding audio frame on the memory 105A.

Subsequently, some streaming processors (SP) 204 fetch video data andaudio data from the memory 105A, and execute instructions which aregiven in advance. Thereby, at last, audio data, in which the voice-cordsrange of a person is emphasized in a scene of the person who isspeaking, is generated by the streaming processors (SP) 204. Then, avideo signal corresponding to video data and an audio signalcorresponding to processed audio data are directly output to theoutside.

The example of processing the audio frames based on the analysis resultof the video frame is not limited to the above-described example. Forexample, the volume of each frame in audio data may be adjusted inaccordance with the luminance of each frame in video data. In this case,the volume of audio data corresponding to a relatively dark scene may bedecreased, and the volume of audio data corresponding to a relativelylight scene may be increased. In addition, a reverberation process or asurround process may be applied to each audio frame belonging to aspecific scene.

In the meantime, in recent years, a so-called “GPGPU” environment, inwhich a GPU is used for general-purpose operations, has been developed.The “GPGPU” environment is used in order to cause the GPU to executegeneral-purpose arithmetic operations such as scientific andtechnological arithmetic operations. In the “GPGPU” environment,instructions and data are sent from the CPU to the GPU, and arithmeticresults obtained by the GPU are sent back to the CPU from the GPU. Thus,even if (1) audio data is input from a CPU 501 to a conventional GPU503, as shown in FIG. 8, (2) a processing result of the audio data needsto be sent back from the GPU 503 to the CPU 501, and further (3) theprocessing result of the audio data needs to be transferred from the CPU501 to a DSP 504. Consequently, many bus bands are consumed.

In the present embodiment, as shown in FIG. 9, the GPU 105 can processboth video data and audio data by using the streaming processors (SP)204, and the GPU 105 can directly output a video signal and an audiosignal. Thus, simply by (1) transferring video data and audio data fromthe CPU 101 to the GPU 105, (2) the GPU 105 can output the video signaland audio signal. Therefore, CPU resources or bus bands may not beconsumed.

Next, referring to FIG. 10, the functional structure of the GPU 105 isdescribed.

The GPU 105 includes a video stream buffer 601, a video decoder 602, avideo frame buffer 603, an audio stream buffer 604, an audio decoder605, an audio frame buffer 606, a video & audio edit process module 607,and a video & audio synchronization process module 608. The video streambuffer 601, video frame buffer 603, audio stream buffer 604 and audioframe buffer 606 are memory areas on the memory 105A.

The video stream buffer 601 is a buffer for storing a stream of videodata which is transferred from the host. The stream of the video datamay be encoded. The audio stream buffer 604 is a buffer for storing astream of audio data which is transferred from the host. The stream ofthe audio data may also be encoded.

The video decoder 602 fetches video data from the video stream buffer601, and decodes the fetched video data. By decoding the video data,frame-unit data, information accompanying each frame (e.g. colorinformation), and a time code for synchronization are extracted. Thecolor information is indicative of a color format (YUV, RGB) of thecorresponding video frame. The video frame (image data), colorinformation and time code, which are obtained on a frame-by-frame basisby the decoding, are stored in the video frame buffer 603. The videodecoder 602 may be realized by causing, for example, at least onestreaming processor (SP) 204 to execute a decoding process. In otherwords, the video decoder 602 may be realized by the arithmetic controlunit 203A and at least one streaming processor (SP) 204 to which thetask for video decoding is allocated by the arithmetic control unit203A.

The audio decoder 605 fetches audio data from the audio stream buffer604, and decodes the fetched audio data. By decoding the audio data,frame-unit data, information accompanying each frame and a time code forsynchronization are extracted. The audio frame (data) and time code,which are obtained on a frame-by-frame basis by the decoding, are storedin the audio frame buffer 606.

The video & audio edit process module 607 executes analysis andprocessing of the video data on the video frame buffer 603, and alsoexecutes analysis and processing of the audio data on the audio framebuffer 606. Besides, the video & audio edit process module 607 can alsoprocess the audio data on the audio frame buffer 606, based on theanalysis result of the frames of the video data on the video framebuffer 603. The video & audio edit process module 607 may be realized bythe arithmetic control unit 203A, at least one streaming processor (SP)204 to which the task for video analysis is allocated by the arithmeticcontrol unit 203A, and at least one streaming processor (SP) 204 towhich the task for audio edit is allocated by the arithmetic controlunit 203A.

The video & audio edit process module 607 can also process each frame ofthe video data on the video frame buffer 603, based on the analysisresult of each frame of the audio data on the audio frame buffer 606.The video & audio edit process module 607 may be realized by thearithmetic control unit 203A, at least one streaming processor (SP) 204to which the task for audio analysis is allocated by the arithmeticcontrol unit 203A, and at least one streaming processor (SP) 204 towhich the task for video edit is allocated by the arithmetic controlunit 203A.

The video & audio synchronization process module 608 synchronizes thevideo data and audio data by using the time code of each analyzed andprocessed video frame and the time code of each analyzed and processedaudio frame. A video signal corresponding to the processed video data isoutput to the outside via the video signal output interface 206. At thesame time, an audio signal corresponding to the processed audio data isoutput to the outside via the audio signal input/output interface 207.

In the above description, the example of processing the video data andaudio data in a cooperative manner has mainly been described. Similarly,each frame of audio data can be processed in accordance with theanalysis result of each frame of graphics data.

As has been described above, according to the present embodiment, theGPU 105 can adaptively apply various processes, which correspond to,e.g. characteristics of each scene of video data, to audio data. Inaddition, the GPU 105 can directly output to the outside the videosignal corresponding to the video data and the audio signalcorresponding to the audio data. Therefore, the audio data can beprocessed in cooperation with the video data.

The various modules of the systems described herein can be implementedas software applications, hardware and/or software modules, orcomponents on one or more computers, such as servers. While the variousmodules are illustrated separately, they may share some or all of thesame underlying logic or code.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

1. A graphics processing unit comprising: a host interface configured toreceive video data and audio data from a host; a memory controllerconfigured to store the received video data and audio data in a memory;a plurality of processing cores coupled to the memory; an arithmeticcontrol unit configured to detect characteristics of the video data byanalyzing frames of the video data on the memory by using at least onefirst processing core of the plurality of processing cores, and to applya process, which is associated with the detected characteristics of thevideo data, to the audio data on the memory, by using at least onesecond processing core of the plurality of processing cores; a videosignal output interface configured to generate a video signalcorresponding to the video data and to output the generated video signalto a display device; and an audio signal output interface configured togenerate an audio signal corresponding to the audio data to which theprocess has been applied, and to output the generated audio signal to asound device.
 2. The graphics processing unit of claim 1, wherein thearithmetic control unit is configured to detect a specific scene in thevideo data, based on the characteristics of the video data, and to applya predetermined process to each of frames of the audio data belonging tothe specific scene.
 3. The graphics processing unit of claim 1, whereinthe arithmetic control unit is configured to allocate to the at leastone first processing core a first task for detecting the characteristicsof the video data by analyzing the frames of the video data on thememory, and to allocate to the at least one second processing core asecond task for applying a process, which is associated with thedetected characteristics of the video data, to the audio data on thememory.
 4. The graphics processing unit of claim 1, wherein thearithmetic control unit is configured to detect characteristics of theaudio data by analyzing frames of the audio data on the memory, by usingat least one processing core of the plurality of processing cores, andto apply a process, which is associated with the detectedcharacteristics of the audio data, to the video data on the memory, byusing at least one processing core of the plurality of processing cores.5. The graphics processing unit of claim 1, further comprising an audiosignal input interface configured to input an audio signal from thesound device.
 6. An information processing apparatus comprising: acentral processing unit (CPU); and a graphics controller coupled to theCPU and configured to process graphics data, video data and audio datain accordance with a request from the CPU, wherein the graphicscontroller comprises: a host interface configured to receive video dataand audio data from the CPU; a memory controller configured to store thereceived video data and audio data in a memory; a plurality ofprocessing cores coupled to the memory; an arithmetic control unitconfigured to detect characteristics of the video data by analyzingframes of the video data on the memory by using at least one firstprocessing core of the plurality of processing cores, and to apply aprocess, which is associated with the detected characteristics of thevideo data, to the audio data on the memory, by using at least onesecond processing core of the plurality of processing cores; a videosignal output interface configured to generate a video signalcorresponding to the video data and to output the generated video signalto a display device; and an audio signal output interface configured togenerate an audio signal corresponding to the audio data to which theprocess has been applied, and to output the generated audio signal to asound device.
 7. The information processing apparatus of claim 6,wherein the arithmetic control unit is configured to detect a specificscene in the video data, based on the characteristics of the video data,and to apply a predetermined process to each of frames of the audio databelonging to the specific scene.
 8. The information processing apparatusof claim 6, wherein the arithmetic control unit is configured toallocate to the at least one first processing core a first task fordetecting the characteristics of the video data by analyzing the framesof the video data on the memory, and to allocate to the at least onesecond processing core a second task for applying a process, which isassociated with the detected characteristics of the video data, to theaudio data on the memory.
 9. A graphics processing unit comprising: aninterface configured to receive video data and audio data; a memorycontroller configured to store the received video data and audio data ina memory; a processor including a plurality of processing cores, theprocessor being coupled to the memory; an arithmetic control unitconfigured to detect characteristics of the video data by analyzingframes of the video data on the memory by using at least a firstprocessing core of the plurality of processing cores, and to apply aprocess, which is associated with the detected characteristics of thevideo data, to the audio data on the memory, by using at least a secondprocessing core of the plurality of processing cores; a video signaloutput interface configured to generate a video signal corresponding tothe video data and to output the generated video signal; and an audiosignal output interface configured to generate an audio signalcorresponding to the audio data to which the process has been applied,and to output the generated audio signal.
 10. The graphics processingunit of claim 9, wherein the arithmetic control unit is configured todetect a specific scene in the video data, based on the characteristicsof the video data, and to apply a predetermined process to each of theframes of the audio data belonging to the specific scene.
 11. Thegraphics processing unit of claim 9, wherein the arithmetic control unitis configured to allocate to at least the first processing core a firsttask for detecting the characteristics of the video data by analyzingthe frames of the video data on the memory, and to allocate to at leastthe second processing core a second task for applying a process, whichis associated with the detected characteristics of the video data, tothe audio data on the memory.
 12. The graphics processing unit of claim9, wherein the arithmetic control unit is configured to detectcharacteristics of the audio data by analyzing frames of the audio dataon the memory, by using at least one processing core of the plurality ofprocessing cores, and to apply a process, which is associated with thedetected characteristics of the audio data, to the video data on thememory, by using at least one processing core of the plurality ofprocessing cores.
 13. The graphics processing unit of claim 9, furthercomprising an audio signal input interface configured to input an audiosignal from a sound device.